Image processing apparatus, method for controlling the same, and image capturing apparatus

ABSTRACT

An image processing apparatus detects predetermined information from an image signal and output a detection signal indicating the detected information, stores the detection signals for a plurality of periods, calculates a first prediction value based on a predetermined number of detection signals, from a newer one to older ones excluding a latest detection signal, stored in the storage unit, calculates a coefficient based on a difference between the first prediction value and a value of the latest detection signal, calculates a second prediction value based on the coefficient and the predetermined number of detection signals, from the latest one to older ones, and generates, based on the second prediction value, a control signal for controlling a predetermined constituent member.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 15/175,849,filed Jun. 7, 2016, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to image processing apparatuses, methodsfor controlling the image processing apparatuses, and image capturingapparatuses, and relates more specifically to an image processingapparatus that performs control based on a signal obtained from an imagesensor provided in an image capturing apparatus, a method forcontrolling the image processing apparatus, and an image capturingapparatus.

Description of the Related Art

Recently, image sensors have become more and more functional, andimprovement in the number of pixels and the frame rate and the like hasbeen achieved. Also, a method for obtaining vector data (optical flow),which indicates movement of an object between a plurality of images,using a signal obtained from an image sensor is known. Moreover, atechnique for combining a plurality of images and detecting a movingobject based on the optical flow is known. Note that, as a method forobtaining the optical flow, the method disclosed in “Digital ImageProcessing”, Computer Graphic Arts Society, pp. 243 to 245 is available.

Meanwhile, time-series signal prediction through signal processing and acontrol method based on prediction have also been proposed. JapanesePatent Laid-Open No. 2013-118450 discloses a signal processing methodwith which a signal at the time of occurrence of an abnormal value isgenerated by a prediction device in a system where the abnormal valueoccurs for a certain time period in an observation signal.

Japanese Patent Laid-Open No. 11-194377 proposes a system for performingimage stabilization by locally performing linear approximation based ona value obtained by camera shake detection. Japanese Patent Laid-OpenNo. 2010-41245 discloses a method for detecting a shake in a focusingdirection, and predicting and controlling a shake that is to occur fromwhen an exposure start instruction is given until exposure is started.

However, with Japanese Patent Laid-Open No. 2013-118450, which relatesto a technique for predicting a future signal from a current observationsignal, a delay that has already occurred in a current observationsignal cannot be compensated.

In Japanese Patent Laid-Open No. 11-194377 and Japanese Patent Laid-OpenNo. 2010-41245, it is assumed that a so-called camera shake is detectedusing some kind of sensor, but a signal obtained after performing imagestabilization is not observed. Since a signal from an image sensor issuitable for detection of the status of an optical system, in the caseof adjusting the optical system before exposure, use of the signal fromthe image sensor enables only the remaining shake after thestabilization to be observed. However, the methods described in JapanesePatent Laid-Open No. 11-194377 and Japanese Patent Laid-Open No.2010-41245 cannot be applied to this kind of system.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and reduces the influence of a delay in adjustment of anoptical system performed based on a signal that is output from an imagesensor.

Furthermore, in a system for adjusting a constituent member of a camerasystem using a signal of an image sensor, the present inventionappropriately adjusts the constituent member even if the signal of theimage sensor cannot be obtained.

According to the present invention, provided is an image processingapparatus comprising: a detection unit configured to detectpredetermined information from an image signal that is output from animage sensor at a predetermined period, and output a detection signalindicating the detected information; a storage unit configured to storethe detection signals for a plurality of periods, from a newer one toolder ones; a first calculation unit configured to calculate a firstprediction value based on a predetermined number of detection signals,from a newer one to older ones excluding a latest detection signal,among the detection signals for the plurality of periods stored in thestorage unit; a computing unit configured to calculate a coefficient forobtaining a second prediction value based on a difference between thefirst prediction value and a value of the latest detection signal; asecond calculation unit configured to calculate the second predictionvalue based on the coefficient and the predetermined number of detectionsignals, from the latest one to older ones, among the detection signalsfor the plurality of periods stored in the storage unit; and ageneration unit configured to generate, based on the second predictionvalue, a control signal for controlling a predetermined constituentmember that is used for obtaining the image signal.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor; and an image processingapparatus that comprises: a detection unit configured to detectpredetermined information from an image signal that is output from theimage sensor at a predetermined period, and output a detection signalindicating the detected information; a storage unit configured to storethe detection signals for a plurality of periods, from a newer one toolder ones; a first calculation unit configured to calculate a firstprediction value based on a predetermined number of detection signals,from a newer one to older ones excluding a latest detection signal,among the detection signals for the plurality of periods stored in thestorage unit; a computing unit configured to calculate a coefficient forobtaining a second prediction value based on a difference between thefirst prediction value and a value of the latest detection signal; asecond calculation unit configured to calculate the second predictionvalue based on the coefficient and the predetermined number of detectionsignals, from the latest one to older ones, among the detection signalsfor the plurality of periods stored in the storage unit; and ageneration unit configured to generate, based on the second predictionvalue, a control signal for controlling a predetermined constituentmember that is used for obtaining the image signal.

Furthermore, according to the present invention, provided is a methodfor controlling an image processing apparatus, the method comprising:detecting predetermined information from an image signal that is outputfrom an image sensor at a predetermined period, and outputting adetection signal indicating the detected information; storing, in astorage unit, the detection signals output in the detecting for aplurality of periods, from a newer one to older ones; calculating afirst prediction value based on a predetermined number of detectionsignals, from a newer one to older ones excluding a latest detectionsignal, among the detection signals for the plurality of periods storedin the storage unit; calculating a coefficient for obtaining a secondprediction value based on a difference between the first predictionvalue and a value of the latest detection signal; calculating the secondprediction value based on the coefficient and the predetermined numberof detection signals from the latest one to older ones, among thedetection signals for the plurality of periods stored in the storageunit; and generating, based on the second prediction value, a controlsignal for controlling a predetermined constituent member that is usedfor obtaining the image signal.

Further, according to the present invention, provided is anon-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of the control methodcomprising: detecting predetermined information from an image signalthat is output from an image sensor at a predetermined period, andoutputting a detection signal indicating the detected information;storing, in a storage unit, the detection signals output in thedetecting for a plurality of periods, from a newer one to older ones;calculating a first prediction value based on a predetermined number ofdetection signals, from a newer one to older ones excluding a latestdetection signal, among the detection signals for the plurality ofperiods stored in the storage unit; calculating a coefficient forobtaining a second prediction value based on a difference between thefirst prediction value and a value of the latest detection signal;calculating the second prediction value based on the coefficient and thepredetermined number of detection signals from the latest one to olderones, among the detection signals for the plurality of periods stored inthe storage unit; and generating, based on the second prediction value,a control signal for controlling a predetermined constituent member thatis used for obtaining the image signal.

Further, according to the present invention, provided is an imageprocessing apparatus comprising: a detection unit configured to detectpredetermined information from an image signal that is output from animage sensor at a predetermined period, and output a detection signalindicating the detected information; a sensor configured to detect aposition signal indicating a position of a predetermined constituentmember, driven based on the detected information, for obtaining theimage signal; and a generation unit configured to generate a controlsignal for controlling the constituent member based on a combined signalobtained by combining the position signal and the detection signal.

Further, according to the present invention, provided is an imageprocessing apparatus comprising: a detection unit configured to detectpredetermined information from an image signal that is output from animage sensor at a predetermined period, and output a detection signalindicating the detected information; a storage unit configured to storethe detection signals for a plurality of periods, from a newer one toolder ones; a prediction unit configured to predict a detection signalthat is to be obtained in a next period based on the detection signalsstored in the storage unit; and a generation unit configured to generatea control signal for controlling a predetermined constituent member forobtaining the image signal based on a latest detection signal if thepredetermined information is detected by the detection unit, andgenerate the control signal based on the detection signal predicted bythe prediction unit if the predetermined information is not detected.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor; and an image processingapparatus that comprises: a detection unit configured to detectpredetermined information from an image signal that is output from theimage sensor at a predetermined period, and output a detection signalindicating the detected information; a sensor configured to detect aposition signal indicating a position of a predetermined constituentmember, driven based on the detected information, for obtaining theimage signal; and a generation unit configured to generate a controlsignal for controlling the constituent member based on a combined signalobtained by combining the position signal and the detection signal.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor; and an image processingapparatus that comprises: a detection unit configured to detectpredetermined information from an image signal that is output from animage sensor at a predetermined period, and output a detection signalindicating the detected information; a storage unit configured to storethe detection signals for a plurality of periods, from a newer one toolder ones; a prediction unit configured to predict a detection signalthat is to be obtained in a next period based on the detection signalsstored in the storage unit; and a generation unit configured to generatea control signal for controlling a predetermined constituent member forobtaining the image signal based on a latest detection signal if thepredetermined information is detected by the detection unit, andgenerate the control signal based on the detection signal predicted bythe prediction unit if the predetermined information is not detected.

Further, according to the present invention, provided is a method forcontrolling an image processing apparatus, the method comprising:detecting predetermined information from an image signal that is outputfrom an image sensor at a predetermined period, and outputting adetection signal indicating the detected information; detecting aposition signal indicating a position of a predetermined constituentmember, driven based on the detected information, for obtaining theimage signal; and generating a control signal for controlling theconstituent member based on a combined signal obtained by combining theposition signal and the detection signal.

Further, according to the present invention, provided is a method forcontrolling an image processing apparatus, the method comprising:detecting predetermined information from an image signal that is outputfrom an image sensor at a predetermined period, and outputting adetection signal indicating the detected information; storing, in astorage unit, the detection signals for a plurality of periods, from anewer one to older ones; predicting a detection signal that is to beobtained in a next period based on the detection signals stored in thestorage unit; and generating a control signal for controlling apredetermined constituent member for obtaining the image signal based ona latest detection signal if the predetermined information is detectedby the detection unit, and generating the control signal based on thepredicted detection signal if the predetermined information is notdetected.

Further, according to the present invention, provided is anon-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of the control methodcomprising: detecting predetermined information from an image signalthat is output from an image sensor at a predetermined period, andoutputting a detection signal indicating the detected information;detecting a position signal indicating a position of a predeterminedconstituent member, driven based on the detected information, forobtaining the image signal; and generating a control signal forcontrolling the constituent member based on a combined signal obtainedby combining the position signal and the detection signal.

Further, according to the present invention, provided is anon-transitory computer-readable storage medium storing a program forcausing a computer to execute the steps of the control methodcomprising: detecting predetermined information from an image signalthat is output from an image sensor at a predetermined period, andoutputting a detection signal indicating the detected information;storing, in a storage unit, the detection signals for a plurality ofperiods, from a newer one to older ones; predicting a detection signalthat is to be obtained in a next period based on the detection signalsstored in the storage unit; and generating a control signal forcontrolling a predetermined constituent member for obtaining the imagesignal based on a latest detection signal if the predeterminedinformation is detected by the detection unit, and generating thecontrol signal based on the predicted detection signal if thepredetermined information is not detected.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing an image capturing apparatus accordingto a first embodiment of the present invention;

FIGS. 2A to 2C are diagrams illustrating a method for calculating anoptical flow;

FIG. 3 is a block diagram showing a configuration of a prediction signalgeneration unit according to a first embodiment;

FIGS. 4A and 4B are diagrams illustrating a sampling delay and specificoperations of the prediction signal generation unit according to thepresent embodiment;

FIGS. 5A and 5B are flowcharts showing operations of a camera systemaccording to the present embodiment;

FIGS. 6A to 6C are external views and a block diagram showing aconfiguration of an image capturing apparatus according to amodification;

FIG. 7 is a block diagram showing a configuration of a prediction signalgeneration unit according to a second embodiment;

FIGS. 8A and 8B are diagrams for illustrating a method for generatingreliability data according to the second embodiment;

FIG. 9 is a block diagram showing a configuration of a prediction signalgeneration unit according to a third embodiment;

FIGS. 10A and 10B are block diagrams showing a configuration of aprediction signal generation unit according to a fourth embodiment;

FIG. 11 is a block diagram showing an image capturing apparatusaccording to a fifth embodiment of the present invention;

FIG. 12 is a block diagram showing a partial schematic configuration ofa camera system control circuit according to a fifth embodiment;

FIG. 13 is a block diagram showing a configuration of a predictionsignal generation unit according to the fifth embodiment;

FIG. 14 is a schematic diagram of a shake signal;

FIGS. 15A to 15C are diagrams illustrating a remaining shake amountafter stabilization at the time of still image shooting according to thefifth embodiment;

FIGS. 16A to 16C are diagrams showing a configuration of an image sensoraccording to a sixth embodiment;

FIG. 17 is a block diagram showing a partial schematic configuration ofa camera system control circuit according to a seventh embodiment;

FIG. 18 is a block diagram showing a configuration of a predictionsignal generation unit according to the seventh embodiment; and

FIGS. 19A to 19C are diagrams illustrating a remaining shake amountafter stabilization at the time of still image shooting according to theseventh embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings.

First Embodiment

FIG. 1 is a block diagram showing a schematic configuration of a camerasystem. The camera system according to a first embodiment is constitutedmainly by a camera body 1 and a lens unit 2 that can be attached to anddetached from the camera body 1. The camera body 1 and the lens unit 2are electrically connected to each other via an electrical contact point11.

The lens unit 2 includes an imaging optical system 3, which isconstituted by a diaphragm and a plurality of lenses including a focuslens and a image stabilization lens that are arranged on an optical axis4, a lens system control circuit 12, and a lens drive unit 13.

Light coming from an object that enters via the imaging optical system 3forms an image on an imaging plane of an image sensor 6 via a shuttermechanism 14. The image sensor 6 performs photoelectric conversion onthe incident light, and outputs an electric signal (image signal) thatcorresponds to a quantity of light.

An image processing unit 7 includes an A/D convertor, a white balanceadjustment circuit, a gamma correction circuit, an interpolationcalculation circuit, a color interpolation processing circuit, and thelike, and can perform given image interpolation processing and colorconversion processing and generate image data for recording and imagedata for display. The image processing unit 7 also compresses image dataand audio data using a predetermined method. Furthermore, the imageprocessing unit 7 performs given calculation processing usingA/D-converted image data, and a camera system control circuit 5 canperform, based on the obtained calculation result, TTL(through-the-lens) autofocusing (AF) processing and auto-exposure (AE)processing. The camera system control circuit 5 thereby obtains theamount of driving of the focus lens included in the imaging opticalsystem 3, the f-number of the diaphragm, and the exposure period of theshutter mechanism 14. The camera system control circuit 5 then notifiesthe lens system control circuit 12 of the amount of driving of the focuslens and the f-number of the diaphragm via the electrical contact point11. The lens system control circuit 12 controls the lens drive unit 13based on the amount of driving of the focus lens and the f-number of thediaphragm of which the lens system control circuit 12 is notified, anddrives the focus lens and the diaphragm. The camera system controlcircuit 5 also controls the shutter mechanism 14 based on the obtainedexposure period.

The image processing unit 7 also includes a shake detection unit 70 thatcompares a plurality of images based on image signals obtained from theimage sensor 6 and generates a shake detection signal. Note thatprocessing in the shake detection unit 70 will be described later indetail.

The image data for recording generated by the image processing unit 7 isoutput to a memory 8 by the camera system control circuit 5. On theother hand, the image data for display is displayed on the display unit9 by the camera system control circuit 5. An electronic view finder(EVF) function can also be achieved by sequentially displaying capturedimage data using the display unit 9. Note that the display unit 9 canarbitrarily turn on and off the display in accordance with aninstruction from the camera system control circuit 5, and if the displayis turned off, power consumption of the camera body 1 can besignificantly reduced.

The camera system control circuit 5 includes a temporary storage areaand a calculation unit, and controls the overall camera system. Inaddition to the aforementioned control, the camera system controlcircuit 5 generates and outputs a timing signal when shooting an image,and controls an imaging system, an image processing system, and arecording/reproduction system in accordance with a user operation madefrom the outside. For example, upon an operation detection unit 10detecting that a shutter release button (not shown) has been pressed,the camera system control circuit 5 controls the driving of the imagesensor 6, image signal processing and compression processing in theimage processing unit 7, recording processing, display content or thelike of information that is displayed on the display unit 9, and thelike. The display unit 9 is a touch panel and is connected to theoperation detection unit 10.

The camera system control circuit 5 includes a prediction signalgeneration unit 50 that predicts a shake based on the shake detectionsignal obtained from the shake detection unit 70 and outputs a shakeprediction signal that indicates the predicted shake. Then, the camerasystem control circuit 5 controls the image stabilization lens via thelens drive unit 13 based on the shake prediction signal indicating thepredicted shake.

FIGS. 2A to 2C are diagrams illustrating an exemplary method forobtaining an optical flow, which is a shake detection signal that isbased on the comparison between a plurality of images performed by theshake detection unit 70. Although a so-called block matching method willbe described in FIGS. 2A to 2C, other methods may alternatively be used.In the present embodiment, it is assumed that the shake detection signalis obtained when displaying a live view and when shooting a movingimage, and image signals that are accumulated and read out from theimage sensor 6 at a predetermined period are used.

Assuming that the latest time of reading out an image signal from theimage sensor 6 is t_(n), FIG. 2A shows an image that is read out at timet_(n−1) in the previous period, and FIG. 2B shows an image that is readout at the time t_(n). FIG. 2C is a diagram in which the images that areread out at the time t_(n−1) and the time t_(n) are overlapped with eachother, and schematically shows a detected vector (shake detectionsignal).

First, in the image that is read out at the time t_(n−1) shown in FIG.2A, attention will be paid to a region 63 in the region of the object62. The size of this region 63 can be arbitrarily set, and may be, forexample, 8×8 pixels. The location in the image obtained at the timet_(n) shown in FIG. 2B to which the region 63 has moved is searched for,with a differential absolute value or the like as a reference.Specifically, in the image in FIG. 2B, a differential absolute value iscalculated while shifting the region as indicated by arrows 66 within apredetermined area, with a region 65 whose position corresponds to theregion 63 as the center. In the case of using the differential absolutevalue as a reference, the position at which the value is smallest is aregion 67 that corresponds to the region 63. As a result, as shown inFIG. 2C, it can be understood that the region 63 shown in FIG. 2A hasmoved between the two images as indicated by a vector 69.

The above operation is performed for a plurality of regions that are setin the screen, and a plurality of moving vectors are detected.Thereafter, one evaluation value of the moving vectors between the twoimages is determined using a method such as selecting a vector whilepaying attention to a main object or obtaining an estimated value with aRANSAC (Random Sample Consensus) method or the like. Note that theRANSAC is a known technique and is not essential part of the inventionof this application, and a description thereof will therefore beomitted. The shake detection unit 70 thus obtains and outputs the shakedetection signal.

FIG. 3 is a block diagram showing an exemplary configuration of theprediction signal generation unit 50 included in the camera systemcontrol circuit 5 according to the first embodiment. In the firstembodiment, the shake prediction signal is generated when displaying alive view and when shooting a moving image. The image signals that areperiodically read out from the image sensor 6 are compared by the shakedetection unit 70 in the image processing unit 7, and a shake detectionsignal u(m) is output and is then input to the prediction signalgeneration unit 50. Note that m in the bracket indicates an m^(th)sampling timing.

The input shake detection signal u(m) is delayed by one period by aplurality of delay units 24(z ⁻¹) that are connected in series. Assumingthat the latest read image signal is an n^(th) sample, the shakedetection signal obtained by delaying the shake detection signal u(n) byone period is a shake detection signal u(n−1), which indicates a shakedetection signal that is based on an image signal sampled in then−1^(th) sampling period. Thereafter, similarly, the shake detectionsignals u(n−2) u(n−M+1), and u(n−M) are defined. These shake detectionsignals u(n), u(n−1), . . . , u(n−M+1), and u(n−M) for a plurality ofperiods are accumulated in a buffer memory (not shown) in the camerasystem control circuit 5. Efficient buffering can be achieved by formingthe buffer memory using, for example, a ring buffer, and replacing dataat a current writing position and changing the writing position everytime a new shake detection signal u(n) is input. Then, by thecalculation denoted by Equation (1), filter coefficients 25 and thebuffered shake detection signals are subjected to convolutionintegration by a series of adders 26. Thereby, a prediction value y(n)of the latest shake detection signal u(n) is obtained from apredetermined number of shake detection signals u(m) up to a shakedetection signal u(n−1) at the previous sampling timing among the shakedetection signals u(m) for a plurality of periods accumulated in thebuffer memory. Note that h_(n−1) indicates the filter coefficients 25that have been obtained until the n−1^(th) sampling timing.

$\begin{matrix}{{y(n)} = {\sum\limits_{m = 1}^{M}{{h_{n - 1}(m)}{u\left( {n - m} \right)}}}} & (1)\end{matrix}$

Equation (1) indicates that time-series prediction is performed by meansof linear combination. It is known that a smooth signal and highlyrepeatable signal can be well approximated by Equation (1). Many signalsfor controlling the imaging optical system 3 have these characteristics.

An error e at this time can be defined by the next Equation (2). Thiserror e can be obtained by subtracting the prediction value y(n)obtained by the series of adders 26 from the latest shake detectionsignal u(n) using an adder 28.e=u(n)−y(n)  (2)

With this definition, a filter for adaptively predicting the next periodusing a known adaptive algorithm in an adaptive algorithm processingunit 29 is formed. As an exemplary adaptive algorithm, an equation forupdating the filter coefficients using an LMS algorithm is expressed byEquation (3). Equation (3) is an exemplary adaptive algorithm, whereasother algorithms such as NLMS and RLS may alternatively be used.h _(n)(i)=h _(n−1)(i)+μeu(i)  (3)

(i=1, 2, . . . , M)

Note that μ in Equation (3) above indicates a step size parameter. Afilter is adaptively formed by repeating the calculation of Equations(1) to (3) for every sampling period and updating h.

With the filter coefficients 30 that are updated as above, a shakeprediction signal y_(p)(n) is obtained by performing convolutionintegration on the shake detection signal u(m) using the series ofadders 26 using the calculation denoted by Equation (4) below.

$\begin{matrix}{{y_{p}(n)} = {\sum\limits_{m = 1}^{M}{{h_{n}(m)}{u\left( {n - m + 1} \right)}}}} & (4)\end{matrix}$

In Equation (4) above, the shake prediction signal y_(p)(n) is a signalof the next sampling timing that is predicted from a predeterminednumber of shake detection signals u(m) obtained until now among theshake detection signals u(m) for the plurality of periods accumulated inthe buffer memory. Equation (4) and Equation (1), which are similar toeach other, are different in the following points. First, filtercoefficients h_(n) that have been updated by Equation (3) are used.Furthermore, the oldest shake detection signal to be used in thecalculation is not the shake detection signal u(n−M) but u(n−M+1), andthe current shake detection signal u(n) is used. As a result, Equation(4) is an equation for calculating a shake prediction signal y_(p)(n) ofthe next sampling period predicted from the signals that have beensampled until now. The prediction signal y_(p)(n) at the next samplingtiming predicted from the latest shake detection signal u(n) is acurrent estimated value from which an accumulation delay has beeneliminated, which will be described later using FIGS. 4A and 4B.

Note that in the above description, the calculation of Equations (1) to(4) is individually performed while paying attention to a certainsampling timing. However, Equation (4) is an equation obtained byreplacing n in Equation (1) with n+1. That is to say, the shakeprediction signal y_(p)(n) can be used as y(n+1) in the next sampling.For this reason, the amount of calculation can be reduced by storing theshake prediction signal y_(p)(n) in the memory. At the next samplingtime, the calculation may be immediately performed in the order ofEquation (2), Equation (3), and Equation (4) while changing the shakeprediction signal y_(p)(n) to y(n+1). Note that, at the first samplingtime, the calculation may be started from Equation (1).

A switch 32 is switched in accordance with later-described tripoddetermination, and in the case where a shake is small, such as whenshooting is performed using a tripod, the switch 32 is controlled so asto output the shake detection signal u(n) as-is, and in other cases, theswitch 32 is controlled so as to output the shake prediction signaly_(p)(n). The output of the switch 32 is used in the control of theimaging optical system 3.

Note that although, in FIG. 3, the switch 32 is switched to output theshake detection signal u(n) when outputting the shake detection signalu(n) as-is in order to facilitate understanding, the same output can beobtained by setting the filter coefficients h_(n)(1)=1, h_(n)(i)=0 (i=2,3, . . . , M). In this case, in the case of outputting the shakedetection signal u(n) as-is as well, the output is generated based onthe shake detection signals u(n−1) to u(n−M+1) that are stored in thebuffer memory. The processing at this time corresponds to processing inthe case where the current observation value is a good prediction value.

Next, a description will be given of a delay due to accumulation, astate of generation of the shake prediction signals, and a difference inoccurrence of an error due to a difference in the sampling period, withreference to FIGS. 4A and 4B.

FIG. 4A is a diagram illustrating an accumulation delay at the time of acertain sampling period, and signal processing according to the firstembodiment. In FIG. 4A, the horizontal axis and the vertical axisrespectively indicate the time and the signal level of the shakedetection signal u(m). A curve 41 indicates an actual shake of thecamera system.

In the case where a shake changes as indicated by the curve 41, in thesystem according to the present embodiment, the shake detection unit 70calculates the shake detection signal u(m) from image signals obtainedby accumulating charges for a predetermined time in the image sensor 6.This is shown in FIG. 4A as a region 44 from time t_(n−5) to timet_(n−4) that corresponds to one sampling period, while assuming that theshake is accumulated. As a shake detection signal u(n−4) based on animage signal that is read out at the next time t_(n−4), a value 45 thatis close to an average value of the shake from the time t_(n−5) to thetime t_(n−4) can be obtained, although it depends on the calculationmethod of the shake detection unit 70.

After sequentially repeating the above processing, a shake detectionsignal u(n+3) based on the image signal that is read out at time t_(n+3)is as shown as a signal value 43. What is to be considered here is thatthe delay is larger than a so-called zero-order hold. In the example inFIG. 4A, since a delay for roughly one sampling period occurs, an error46 occurs between the actual shake 41 and the detected shake detectionsignal u(n+3) indicated by the signal value 43. Since this error 46 isan error due to a delay, the error 46 becomes large at a point where thedifferential value of the signal is large. Considering the frequencyspace, a frequency component regarding which a sampling delay cannot beignored appears as an error.

FIG. 4B shows the case of observing the same actual shake 41 with ashorter sampling period. At this time as well, the shake detectionsignal u(m) indicated by a signal value 55 is detected in a region 54,and the shake detection signal u(m) indicated by a signal value 53 isdetected by repeating this. An error 56 at this time is obviouslysmaller than the error 46 in FIG. 4A. Thus, whether or not an error dueto the sampling can be ignored is determined by the amount of delaydetermined by the frequency band (the band that is desired to becontrolled, which is, here, the frequency of the shake) included in thesignal that is to be observed, and the sampling period. That is to say,if the sampling period is sufficiently short with respect to the bandthat is desired to be controlled (in general, if the sampling frequencyis about ten times the band that is desired to be controlled), theinfluence of the delay due to sampling can be ignored. However, thecurrent sampling period for the image sensor 6 is usually about 60frames per second, and cannot be regarded as a sufficient samplingperiod when considering a signal obtained when a camera shake occurs,for example, and the delay cannot be ignored.

Returning to FIG. 4A, compensation of a sampling delay based onprediction will be described. First, processing at the time t_(n) willbe considered. When M in Equations (1) to (4) described with referenceto FIG. 3 is 3, the shake detection signals u(n), u(n−1), u(n−2), andu(n−3) that are obtained respectively at the time t_(n), t_(n−1),t_(n−2), and t_(n−3) are stored in the buffer memory. The calculation ofEquation (1) is performed using these signals. In FIG. 4A, thiscalculation is schematically shown with lines that are drawn downward.That is to say, the prediction value y(n) is calculated from the shakedetection signals u(n−1), u(n−2), and u(n−3) and filter coefficientsh_(n−1)(1), h_(n−1) (2), and h_(n−1) (3).

Thereafter, the filter coefficients h are updated by the adaptivealgorithm processing unit 29, and the filter coefficients h_(n)(1),h_(n)(2), and h_(n)(3) are obtained. Then, the shake prediction signaly_(p)(n) is calculated based on Equation (4). In FIG. 4A, thiscalculation is schematically shown with lines that are drawn upward.That is to say, the shake prediction signal y_(p)(n) is calculated fromthe shake detection signals u(n), u(n−1), and u(n−2) and the filtercoefficients h_(n)(1), h_(n)(2), and h_(n)(3). The value of the shakeprediction signal y_(p)(n) that is thus obtained is as indicated by asignal value 47 denoted by a dotted line.

As described using the region 44 and the signal value 45, the shakedetection signal u(m) detected by the shake detection unit 70 delays forabout one sampling period. For this reason, the shake prediction signaly_(p)(n), which is a signal that is predicted from the signals obtaineduntil now at the time t_(n+1) corresponding to the next sampling timing,is used as an estimated value of the shake detection signal at thecurrent time t_(n). That is to say, the prediction value of the signalat the next sampling timing is used as the current estimated value.Since a future observation value u(n+1) is not used for calculating theshake prediction signal y_(p)(n), the calculation can be performedwithout inconsistency. In a system where an accumulation delay occurs asshown in FIG. 4A, the difference between the shake prediction signaly_(p)(n) of the signal at the next sampling timing and the actual shake41 is small. The difference is small not only at the time t_(n) but alsoat other time. That is to say, it can be understood that the predictionvalue of the signal of the next sampling is a good current estimatedvalue.

As a result, it is possible to reduce the influence of the accumulationdelay in the image sensor 6 and adjust the imaging optical system 3based on the signals of the image sensor 6. Since the influence of theaccumulation delay is small, more appropriate image stabilizationcontrol can be performed, and a higher-quality image with a reducedshake can be obtained.

Next, camera system control according to the present embodiment will bedescribed with reference to flowcharts in FIGS. 5A and 5B. FIG. 5A showsthe overall flow of the operations of the camera system, andparticularly shows processing associated with the present invention.FIG. 5B shows exposure/optical control including the image stabilizationcontrol.

In FIG. 5A, upon the processing being started by, for example, turningon the power of the camera system, in step S110, the filter of theprediction signal generation unit 50 is reset. In this step S110, thebuffered shake detection signals u(n), u(n−1), . . . , u(n−M+1), andu(n−M) and the filter coefficients h₀(1), h₀(2), . . . , h₀(M) shown inFIG. 3 are replaced with predetermined values. Here, for example, allvalues are set to zero.

In step S120, it is determined whether or not the camera system is setin a fixed manner. For example, the shake will be small if the camerasystem is fixed as in the case of being set on a tripod, and therefore,the amount of calculation can be reduced by omitting the shake detectionmethod according to the present embodiment. For example, if the absolutevalue of the shake detection signal u(m) detected by the shake detectionunit 70 consecutively falls below a predetermined threshold value duringa given period (i.e., if the shake detection signal u(m) stays within apredetermined range), it is determined that the camera system is set ona tripod or the like. Then, a flag indicating it is set.

In step S130, the turning off of the power is monitored. If the power isturned off, the operation is stopped and the processing ends, and ifnot, the processing proceeds to step S140. In step S140, exposure andimage stabilization control are performed. Note that the specificprocessing in step S140 will be described later with reference to FIG.5B.

In step S150, an image obtained by shooting is displayed. This isbecause the processing in steps S140 and S150 is performed during ashooting preliminary operation, and in step S150, image is not recorded,and an image is displayed on the display unit 9 provided in the camerabody 1. The user can thereby check the composition, for example.

In step S160, a start of recording is determined. If an instruction tostart recording is given by a user operation, the processing proceeds tostep S170, and if not, the processing returns to step S120 and theshooting preliminary operation is repeated.

In step S170, the buffer memory is reset, and the image stabilizationlens is centered. With this operation, the image stabilization lens islocated at a position close to the center of the optical axis and canmove in any direction to the same degree, and therefore, the probabilitythat the image stabilization lens reaches the limit of the driving rangedecreases, and image quality can be improved and image stabilizationrange can be enlarged.

In step S180, the same processing as step S140, which is performed inthe shooting preliminary operation, is performed. However, as will bedescribed later, a sensor reading method may be changed so as to beappropriate for each mode. In step S190, a shot image is displayed andrecorded. The shooting is performed by repeating steps S180 and S190.These steps are called a “shooting operation” here.

In step S200, the end of recording is determined. If an instruction toend the recording is given by a user operation, the processing proceedsto step S120 and returns to the shooting preliminary operation. If not,the processing returns to step S180 and the shooting operation isrepeated.

Next, the exposure/optical control will be described with reference toFIG. 5B. The exposure/optical control shown in FIG. 5B is called fromboth step S140 during the shooting preliminary operation and step S180during the shooting operation.

Upon the processing being called, in step S310, an image signal is readout from the image sensor 6. Simultaneously, reset is performed, andaccumulation up to the next readout is started. Note that the readout instep S310 may be performed using methods appropriate respectively forthe shooting preliminary operation and the shooting operation. Forexample, if the image to be output in the shooting preliminary operationis output to the display unit 9 provided in the camera body 1, an imagefitted to the resolution of the display unit 9 need only be obtained,and accordingly, power consumption can be suppressed by performingthinning readout. Furthermore, in the shooting operation, the resolutionand the readout range (so-called crop shooting) are changed based onuser settings.

Upon the image signal being read out in step S310, the imagestabilization control using the read image signal is started. In stepS320, the shake detection unit 70 calculates the optical flow asdescribed above, and obtains the shake detection signal u(m). In stepS330, whether or not the camera system is set on a tripod is determinedbased on the flag that is set in step S120. If it is determined that thecamera system is set on a tripod, the processing proceeds to step S340,and through-output is performed. The “through-output” mentioned heremeans that the shake detection signal u(m) detected by the shakedetection unit 70 is output as-is as the signal to be used in the imagestabilization control. This corresponds to switching the switch 32 inFIG. 3 to use the input shake detection signal u(m) as-is as the controlsignal.

On the other hand, if the camera system is not set on the tripod, theprocessing proceeds to step S350, and the shake prediction signal iscalculated and output. Here, the shake prediction signal y_(p) that hasbeen processed by the filter shown in FIG. 3 is output as the controlsignal to be used in the image stabilization control. This correspondsto switching the switch 32 in FIG. 3 to use the shake prediction signaly_(p)(n) as the control signal.

In step S360, the position of the image stabilization lens is controlledbased on the control signal obtained in step S340 or S350, therebydriving the imaging optical system 3. After the above processing ends,the processing returns to the processing in FIG. 5A.

As described above, according to the first embodiment, the influence ofthe delay can be reduced in the case of adjusting the imaging opticalsystem 3 using the image signal of the image sensor in which anaccumulation delay occurs. As a result, the influence of theaccumulation delay is small, and accordingly, more appropriate imagestabilization control is performed, and the picture quality can beimproved.

Note that although the above example has described the case ofperforming the image stabilization control by driving the imagestabilization lens included in the imaging optical system 3, the imagestabilization control can also be performed by driving the image sensor6.

Furthermore, although the above example has described the case ofpredicting a shake, the present invention is not limited to the shakeprediction and the image stabilization lens control, and is applicableto various kinds of control of each part of the camera system using theimage signal that is output from the image sensor. For example, aconfiguration is conceivable in which the prediction value is obtainedusing a signal indicating a focus state of an image in the case ofperforming focus adjustment using imaging plane phase difference AF, andusing a signal indicating luminance information that indicates thebrightness of an image in order to control exposure, thereby controllingthe focus lens, the diaphragm, the shutter speed, and the like.

Modification

A modification of the present invention is shown in FIGS. 6A to 6C.FIGS. 6A to 6C are diagrams showing an image capturing apparatus 100according to the modification, and show the case where a lens unit and acamera body are integrally configured. FIGS. 6A and 6B are perspectiveviews of the appearance of the image capturing apparatus taken from thefront and the back, and FIG. 6C is a block diagram showing an electricalconfiguration thereof. In this image capturing apparatus, the boundarybetween the lens unit and the camera body is not clear, and do notindividually have a system control unit in many cases. In FIG. 6C, thecamera system control circuit 105 controls the imaging optical system 3by directly controlling the lens drive unit 13. Note that elementshaving functions similar to those in FIG. 1 will be given the samereference numerals, and a description thereof will be omitted.

In the image capturing apparatus 100 having the configuration shown inFIG. 6A to 6C as well, processing similar to that in the firstembodiment can be performed.

Second Embodiment

Next, a second embodiment of the present invention will be described.FIG. 7 is a block diagram showing a configuration of a prediction signalgeneration unit 50′ according to the second embodiment, and theprediction signal generation unit 50′ can be used in place of theprediction signal generation unit 50 shown in FIG. 1 or 6C. Note thatsince the elements other than the prediction signal generation unit 50′are similar to those shown in FIG. 1 or 6C, a description thereof willbe omitted here.

A difference between the prediction signal generation unit 50′ shown inFIG. 7 and the prediction signal generation unit 50 shown in FIG. 3 liesin further having a comparator 35 and a switch 36. The other elementsare similar to those shown in FIG. 3, and accordingly, the samereference numerals will be assigned thereto and a description thereofwill be omitted.

In the second embodiment, the shake detection unit 70 also obtains dataregarding reliability in addition to the shake detection signal u(m).Note that a method for acquiring the data regarding reliability will bedescribed later. The reliability data is input to the comparator 35. Ifit is determined that the obtained reliability is lower thanpredetermined reliability, the switch 36 is operated, and the input tothe buffer memory is switched from the shake detection signal u(n) ofthe shake detection unit 70 to the prediction value y(n) at the previoussampling timing obtained by the prediction signal generation unit 50.

If the adaptive filter is appropriately formed, the prediction valuey(n) is to be a good prediction value of the shake detection signalu(n). With a small number of samples, the signal will not beinconsistent even if the prediction value y(n) is used in place of theshake detection signal u(n). For this reason, in the case where, forexample, the shake detection unit 70 fails the detection, moreappropriate processing than using a less reliable shake detection signalcan be performed.

Here, a description will be given, using FIGS. 8A and 8B, of a situationwhere the reliability lowers, and the method for generating thereliability data. FIG. 8A is a diagram schematically showing a state ofacquiring images in time series from a plurality of visual points. Anarrow 79 indicates a light beam from a light source, and 80 denotes anobject. As an example, FIG. 8A schematically shows that images areobtained from three visual points indicated by arrows 81, 82, and 83. Asshown in FIG. 8A, when shooting is performed from the plurality ofvisual points 81, 82, and 83, object images 84, 85, and 86 of the sameobject 80 are formed at different positions in the respective images.Note that 84 a, 85 a, and 86 a respectively indicate positions of mirrorreflection components on the object images 84, 85, and 86.

When the optical flow is obtained from the object images 84, 85, and 86by the shake detection unit 70, and these images are overlapped suchthat the object 80 is in the same position, an image shown in FIG. 8B isobtained. That is to say, FIG. 8B shows a state where the object images84, 85, and 86 overlap one another, and conversely, frames 81 a, 82 a,and 83 a, which indicate image object acquisition areas, are shifted. 87denotes a frame that indicates a partial region to be used forpositioning.

In the image in FIG. 8B, the mirror reflection components 84 a, 85 a,and 86 a are located at different positions, and with such an object,the degree of coincidence of the images lowers. In a situation where thedegree of coincidence of the images lowers, the values of the opticalflow determined based on the mirror reflection component differs fromthe optical flow determined based on a diffuse reflection component.Which of these values to be selected depends on the signal intensity ofthe mirror reflection component and the diffuse reflection component,the size of the region of interest at the time of calculating theoptical flow, and the like. That is to say, it is expected that theoptical flow is instable.

Therefore, such a degree of coincidence of the images can be used asreliability data. If the degree of coincidence of the images is low,reliability data that indicates low reliability is output. As anothermethod, the degree of coincidence of the images may be madedimensionless by dividing the degree of coincidence of the images by thebrightness or the rate of change of the degree of coincidence.

As described above, according to the second embodiment, more appropriateimage stabilization control can be performed even in the case where thereliability of the shake detection signal obtained from the shakedetection unit is low.

Third Embodiment

Next, a third embodiment of the present invention will be described.FIG. 9 is a block diagram showing a configuration of a prediction signalgeneration unit 50″ according to the third embodiment, and theprediction signal generation unit 50″ can be used in place of theprediction signal generation unit 50 shown in FIG. 1 or FIGS. 6A to 6C.Note that since the elements other than the prediction signal generationunit 50″ is similar to those shown in FIG. 1 or FIGS. 6A to 6C, adescription thereof will be omitted here.

The prediction signal generation unit 50″ shown in FIG. 9 is similar tothe prediction signal generation unit 50′ shown in FIG. 7, but isdifferent in that a continuity determination device 37 is provided inplace of the comparator 35. A shake detection signal u(m) optical flow,which is the output of the shake detection unit 70, is input to thecontinuity determination device 37. In the case of periodicallyobserving the movement of an object, it is hard to consider that thevalue of the shake detection signal u(m) suddenly changes beyond athreshold value. That is to say, it is hard to consider that an objectwhich moves from one end to the other end of the screen during a periodis a main object. Therefore, in the example in FIG. 9, the continuitydetermination device 37 determines the reliability based on thetime-series output of the shake detection unit 70. If the value of theshake detection signal u(m) changes beyond the threshold value, it isdetermined that the reliability is low, then the switch 36 is operatedto switch the input to the buffer memory from the shake detection signalu(m) from the shake detection unit 70 to the prediction value y(n) atthe previous sampling timing of the prediction signal generation unit50″.

As described above, according to the third embodiment, more appropriatecontrol can be performed even in the case where the reliability of theshake detection signal obtained from the shake detection unit is low.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be described.Another switching control of the switch 32 will be described using FIGS.10A and 10B. FIGS. 10A and 10B are block diagrams showing aconfiguration of the prediction signal generation unit 50′, which issimilar to the configuration shown in FIG. 7. However, delay units 24that are shaded in FIGS. 10A and 10B indicate that the immediatelyprevious buffer content is the prediction value y(n) of the previoussampling timing obtained by the prediction signal generation unit 50.That is to say, in FIG. 10A, although the shake detection signals u(n),u(n−2), and u(n−M+2) were to be buffered in the case where thereliability of the shake detection signals is high, the predictionvalues y(n), y(n−2), and y(n−M+2) st the respective previous samplingtimings have been buffered. Also, FIG. 10B shows that the predictionvalues y(n), y(n−1), and y(n−2) have been buffered in place of the shakedetection signals u(n), u(n−1), and u(n−2).

FIG. 10A is a schematic diagram showing the case where the replacementwith the prediction value y(n) at the previous sampling timing is notconsecutive but the occurrence frequency thereof has reached a fixedratio, and FIG. 10B shows the case where the replacement with theprediction value y(n) at the previous sampling timing has consecutivelyoccurred.

As shown in FIG. 10A, a value buffered at the M+1^(th) sampling timingis referenced in order to generate the current shake prediction signaly_(p)(n). If many of the referenced values are signals of the predictionvalue y(n) of the previous sampling timing, the shake detection signalu(n) of the shake detection unit 70 is not referenced and almost onlythe prediction values y(n) are used, and then, there is a possibilitythat the shake prediction signal y_(p)(n) will not be a sufficientlygood prediction value. Therefore, if the number of prediction valuesy(n) stored in the buffer memory exceeds a predetermined number, theswitch 32 is operated, and the image stabilization lens is operatedbased on the signal of the shake detection unit 70. This is because,even if the signal of the shake detection unit 70 is affected by adelay, it is considered to be better than the prediction value y(n).

Similarly, if the replacement with the prediction value y(n)consecutively occurs as in FIG. 10B, it is also conceivable that thequality of the shake prediction signal y_(p)(n) rapidly lowers.Therefore, if the number of the prediction values y(n) stored in thebuffer memory consecutively exceeds the predetermined number, the switch32 is operated, and the image stabilization lens is operated based onthe shake detection signal u(n) from the shake detection unit 70,similarly as in the case of FIG. 10A.

Fifth Embodiment

Next, a fifth embodiment of the present invention will be described.FIG. 11 is a block diagram showing a schematic configuration of a camerasystem according to the fifth embodiment. Note that the camera systemaccording to the fifth embodiment is the camera system described in thefirst embodiment with reference to FIG. 1 in which the shutter mechanism14 is not provided and to which a position sensor 15 is added.Accordingly, elements similar to those in FIG. 1 will be given the samereference numerals, and a description thereof will be omitted asappropriate.

A lens unit 2′ according to the fifth embodiment includes an imagingoptical system 3 constituted by a diaphragm (constituent member) and aplurality of lenses including a focus lens and a image stabilizationlens (constituent member) that are arranged on an optical axis 4, a lenssystem control circuit 12, a lens drive unit 13, and a position sensor15. The position sensor 15 detects the positions of the focus lens andthe image stabilization lens, and outputs respective lens positionsignals.

Note that, in the fifth embodiment, the camera system control circuit 5obtains drive data for driving the image stabilization lens throughlater-described processing based on a shake detection signal obtainedfrom the shake detection unit 70 and the amount of lens movement of theimage stabilization lens, and transmits the drive data to the lens unit2′.

In the fifth embodiment as well, as described above with reference toFIGS. 2A to 2C, an optical flow, which is a shake detection signal basedon the comparison between a plurality of images, is obtained by theshake detection unit 70.

In the processing described with reference to FIG. 2, if an object isstationary, movement of an image occurs due to a camera shake, andtherefore the camera shake is detected. If the object is moving, a shakethat is a combination of the movement of the object and a camera shakeis detected.

FIG. 12 is a block diagram showing an exemplary configuration forperforming image stabilization according to the fifth embodiment that isincluded in the camera system control circuit 5.

In the configuration shown in FIG. 12, the operation is switched basedon a Reset signal, a signal S0, a signal S1, and a signal S2. Forexample, the Reset signal is output at the time when, for example, thepower of the camera system is turned on or a lens is replaced. Thesignal S0 is a signal that is output when an instruction not to performthe image stabilization processing is given, and the signal S1 is asignal that is output when an instruction to perform the imagestabilization processing is given. The signal S2 is a signal that isoutput when in a state where the image stabilization processing cannotbe performed when the instruction to perform the image stabilizationprocessing has been given. For example, the signal S0 is output if theimage stabilization processing is turned off in accordance with a useroperation, the signal S1 is output if the image stabilization processingis turned on, and the signal S2 is output if the shutter release button(not shown) is pressed in this state. Furthermore, in addition to theuser operation, the control may be performed such that the signal S0 isoutput if the camera system is fixed to a tripod or the like and it isdetermined that the image stabilization processing is not necessary, andthe signal S1 is output if it is determined that the image stabilizationprocessing is necessary. Note that the cases where the Reset signal, thesignal S0, the signal S1, and the signal S2 are output are not limitedto the above example, and can be set as appropriate.

In FIG. 12, a gain adjustment unit 230 obtains the amount of lensmovement by comparing a plurality of lens position signals that areoutput from the position sensor 15 and acquired at different time, andconverts the obtained amount of the lens movement into the amount ofchange of the angle of view in the image sensor 6. The relationshipbetween the amount of lens movement and the amount of change of theangle of view in the image sensor 6 depends on characteristics of theoptical system, and the amount of change of the angle of view in theimage sensor 6 is obtained based on this information and output to theadder 222.

The adder 222 adds the amount of change of the angle of view obtainedfrom the gain adjustment unit 230 and the shake detection signal fromthe shake detection unit 70. By this addition, the camera shake amountu(m) (combined signal) in the case where the image stabilizationprocessing is not performed, i.e., in the case where the center of theimage stabilization lens is located at a reference position thatcoincides with the optical axis can be obtained.

The prediction signal generation unit 223 is for predicting the currentcamera shake amount based on the input camera shake amount u(m) andoutputting the shake prediction signal indicating the predicted camerashake amount, and the operation is switched based on the Reset signal,the signal S0, the signal S1, and the signal S2. Note that the internalconfiguration and specific operations of the prediction signalgeneration unit 223 will be described later with reference to FIG. 13.

A limiter 224 sets an upper limit and a lower limit of the shakeprediction signal obtained from the prediction signal generation unit223 and restricts the shake prediction signal. In general, predictionaccuracy in processing based on prediction lowers with the lapse oftime, and therefore, the upper limit and the lower limit of the shakeprediction signal from the prediction signal generation unit 223 is setin advance. In this case, there are possible methods such as settingpredetermined upper and lower limits of the shake prediction signal,setting the upper limit and the lower limit of the shake predictionsignal in accordance with the time that has elapsed since the signal S2was input, or setting the upper limit and the lower limit of the shakeprediction signal in accordance with the charge accumulation time. Then,processing for cutting off, at the upper limit or lower limit, the shakeprediction signal that exceeds the set upper limit or falls below theset lower limit is performed. Note that the limiter 224 is not anessential element in the invention of the present application.

A switch 225 is subjected to switching control by the signal S0, thesignal S1, and the signal S2. If the signal S0 or the signal S1 isinput, the signal of the adder 222 is selected, and if the signal S2 isinput, the signal of the limiter 224 is selected. The control unit 226compensates the gain and the phase of the signal selected by the switch225 such that the feedback system is stabilized.

A switch 227 is subjected to switching control by the signal S0, thesignal S1, and the signal S2. If the signal S0 is input, the switch 227is grounded to a ground portion 228. Thus, the driving of the imagingoptical system 3 is stopped, and the image stabilization is notperformed. If the signal S1 or the signal S2 is input, the switch 227 isswitched to be connected to the control unit 226, and the imagestabilization lens is driven in accordance with a control signal fromthe control unit 226.

With the above configuration and control, a ground level signal isoutput as the control signal for the image stabilization lens if thesignal S0 is input, and a signal based on the camera shake amount u(m)that is output from the adder 222 is output as the control signal if thesignal S1 is input. If the signal S2 is input, a signal based on thecamera shake amount estimated by the prediction signal generation unit223 is output as the control signal for the image stabilization lens.

Next, the configuration and processing of the prediction signalgeneration unit 223 will be described. FIG. 13 is a block diagramshowing an exemplary configuration of the prediction signal generationunit 223 included in the camera system control circuit 5 according tothe fifth embodiment. As described above, the camera shake amount u(m)that is output from the adder 222 is input to the prediction signalgeneration unit 23. Note that m in the bracket indicates an m^(th)sampling timing.

The input camera shake amount u(m) is delayed by one period by aplurality of delay units 231(z ⁻¹) that are connected in series.Assuming that the latest read image signal is an n^(th) sample, thecamera shake amount obtained by delaying the camera shake amount u(n) byone period is a camera shake amount u(n−1), which indicates a camerashake amount that is based on an image signal sampled in the n−1^(th)sampling period. Thereafter, similarly, the camera shake amounts u(n−2),. . . , u(n−M+1), and u(n−M) are defined. These camera shake amountsu(n), u(n−1), . . . , u(n−M+1), and u(n−M) for a plurality of periodsare accumulated in a buffer memory (not shown) in the camera systemcontrol circuit 5. Efficient buffering can be achieved by forming thebuffer memory using, for example, a ring buffer, and replacing data at acurrent writing position and changing the writing position every time anew camera shake amount u(n) is input. Then, by the calculation denotedby Equation (1), filter coefficients 232 and the buffered camera shakeamounts are subjected to convolution integration by a series of adders233. Thereby, a prediction value y(n) of the latest camera shake amountu(n) is obtained from a predetermined number of camera shake amountsu(m) up to the camera shake amount u(n−1) at the previous samplingtiming, of the camera shake amounts u(m) for the plurality of periodsaccumulated in the buffer memory. Note that h_(n−1) indicates the filtercoefficients 232 that has been obtained until the n−1^(th) samplingtiming.

$\begin{matrix}{{y(n)} = {\sum\limits_{m = 1}^{M}{{h_{n - 1}(m)}{u\left( {n - m} \right)}}}} & (1)\end{matrix}$

Equation (1) indicates that time-series prediction is performed bylinear combination. It is known that a smooth signal and a highlyrepetitive signal can be well approximated in the form like Equation(1). Many signals for controlling the imaging optical system 3 havethese characteristics.

An error e at this time can be defined by the next Equation (2). Thiserror e can be obtained by subtracting the prediction value y(n)obtained by the series of adders 233 from the latest camera shake amountu(n) using an adder 234.e=u(n)−y(n)  (2)

With this definition, a filter for performing adaptively predicting thenext period using a known adaptive algorithm in an adaptive algorithmprocessing unit 235 is formed. As an exemplary adaptive algorithm, anequation for updating the filter coefficients using an LMS algorithm isexpressed by Equation (3). Equation (3) is an exemplary adaptivealgorithm, whereas other algorithms such as NLMS and RLS mayalternatively be used.h _(n)(i)=h _(n−1)(i)+μeu(i)  (3)

(i=1, 2, . . . , M)

Note that μ in Equation (3) above indicates a step size parameter. Afilter is adaptively formed by repeating the calculation of Equations(1) to (3) for every sampling period and updating h.

A switch 237 is switched by the Reset signal. If the reset signal isoutput, the switch 237 is grounded to the ground portion 236, and thefilter coefficients 232 are reset. That is to say, a filter coefficientsh_(n−1) are initialized. If the Reset signal is canceled, the switch 237is connected to the adaptive algorithm processing unit 235, and thefilter coefficients 232 are updated. Even if the characteristics of theshake change as in the case where a different photographer shoots animage or due to lens replacement or the like, the filter coefficients232 can be initialized by performing the reset operation.

A switch 238 is switched in accordance with the signal S0, the signalS1, and the signal S2. The switch 238 is closed if the signal S0 and thesignal S1 are input, and the switch 238 is opened if the signal S2 isinput. Thereby, the filter coefficients 232 are updated if the signal S0and the signal S1 are input, and the updating of the filter coefficients232 is stopped if the signal S2 is input.

Then, the camera shake amount u(m) is delayed by delay units 231, andconvolution integration is performed by the series of adders 233 withthe calculation expressed by the next Equation (5) using the above-setfilter coefficients 232, thereby obtaining a prediction value y(n+1).

$\begin{matrix}{{y\left( {n + 1} \right)} = {\sum\limits_{m = 1}^{M}{{h_{n}(m)}{u\left( {n - m + 1} \right)}}}} & (5)\end{matrix}$

In Equation (5) above, the right-hand side is the value obtained untilthe n^(th) sampling, and the left-hand side is the prediction value ofthe camera shake amount at the n+1^(th) sampling timing.

A switch 239 is switched in accordance with the signal S0, the signalS1, and the signal S2, similarly to the switch 238, and the switch 239enters a state of sending the input camera shake amount u(n) to a laterstage if the signal S0 or the signal S1 is input. If the signal S2 isinput, the switch 239 inputs the aforementioned prediction value y(n+1)as the camera shake amount at the next sampling timing. Thus, thecoefficient obtained by the adaptive algorithm in the adaptive algorithmprocessing unit 235 is used, and this error is small if y(n+1) is a goodprediction value of the camera shake amount u(n+1), then the calculationis recursively performed using y(n+1) in place of the camera shakeamount u(n+1). Since the shake detection signal cannot be obtained in anexposure state after the signal S2 is input, the prediction value issimilarly obtained with recursive calculation with the filter shown inFIG. 13 until the signal S0 or the signal S1 is input.

Thus, with the prediction signal generation unit 223 shown in FIG. 13,the adaptive filters are updated based on the camera shake amount u(m)that is obtained while the signal S0 or the signal S1 is input, and theprediction value y(n+1) after the signal S2 is input based thereon isoutput.

Note that although the prediction signal generation unit 223 shown inFIG. 13 uses an adaptive filter, signals at several sampling timingsimmediately before the signal S2 is input may be linearly approximatedand moved along this line. This is because the error is small even iflinear approximation is performed in the case of a very short exposureperiod.

Next, a signal detected before and after the image stabilization isstarted will be described using FIG. 14. In FIG. 14, the horizontal axisindicates the time, and the vertical axis indicates the actual camerashake amount, the image stabilization amount, and the remaining shakeamount after stabilization. The remaining shake amount afterstabilization is defined as the difference between the actual camerashake amount and the image stabilization amount, and corresponds to theshake detection signal detected by the shake detection unit 70. Here,the case is shown where the signal S0 is changed to the signal S1 attime t_(on) near the center in FIG. 14, the image stabilizationprocessing is changed from an off state to an on state, and the imagestabilization processing is started.

In a state where the signal S0 is input before the time t_(on), theimage stabilization processing is not performed. Accordingly, the imagestabilization amount is 0, and the camera shake amount coincides withthe remaining shake amount.

In a state where the signal S1 is output after the time t_(on), theimage stabilization processing is performed, the image stabilizationamount has a waveform similar to the camera shake amount, and theremaining shake amount has a waveform with a shape different from thecamera shake amount, as shown in FIG. 14. Here, the image stabilizationamount does not completely coincide with the camera shake amount due tothe influence of the control system such as a delay in following, sincethe image stabilization amount is the amount of control based on thesignal that is read out by the image sensor 6, for example.

If the shake is stabilized by the image stabilization lens included inthe imaging optical system 3, the signal characteristics greatly changebefore and after the time t_(on) at which the signal S1 is output. Here,the image stabilization amount indicated by a dotted line in FIG. 14 canbe obtained using a lens position signal of the image stabilization lensobtained from the position sensor 15. As mentioned above, the camerashake amount indicated by a thick solid line itself can be detected evenafter the signal S1 is output, by adjusting the signal of the positionsensor 15 using the gain adjustment unit 230 and adding the adjustedsignal to the shake detection signal obtained by the shake detectionunit 70. The characteristics of the camera shake amount do not changebefore and after the signal S1 (OFF state→ON state of the imagestabilization processing), which is convenient for performing theprediction.

As described in FIG. 13, the prediction signal generation unit 223stores the frequency characteristics of the camera shake amount u(m) asthe filter coefficients using the adaptive filter. If the signalcharacteristics change, the filter coefficients also need to be greatlyupdated, whereas, with the configuration according to the fifthembodiment, the signal characteristics do not change before and afterthe signal S1 due to the adder 222, and it is therefore possible tocontinue the prediction using the filter coefficients that has been usedthus far.

This control is particularly effective in the case where, with the startof the image stabilization processing, the signal S2 is outputimmediately after the signal S1 is output (e.g., exposure in the stillimage shooting is started). That is to say, in the case of performingthe prediction using the remaining shake amount after stabilization(shake detection signal), if the characteristics of the remaining shakeamount after stabilization greatly change due to the signal S1, thefilter coefficients need to be updated, but if the signal S2 is outputbefore the updating ends, there is a possibility that the predictionbecomes inappropriate. On the other hand, with the control according tothe fifth embodiment, even if the signal S2 is output immediately afterthe signal S1 is output, the prediction will not become inappropriate.

FIGS. 15A to 15C are diagrams for illustrating the camera shake amountand the remaining shake amount after stabilization in the case ofperforming the prediction processing and in the case of not performingthe prediction processing. In order to facilitate understanding of thedescription, it is assumed in FIGS. 15A to 15C that the actual shakeapplied to the camera system is a simple sine wave. Furthermore, it isassumed here that the signal S0, the signal S1, and the signal S2 areswitched by an operation made on a shutter button (not shown) whenshooting a still image. Specifically, the signal S0 is output in a statewhere the shutter button is not pressed, the signal S1 is output in astate where the shutter button is half-pressed and an instruction toprepare for shooting has been given, and the signal S2 is output in astate where the shutter button is full-pressed and an instruction torecord an image has been given.

FIG. 15A is a graph showing the case of performing the linear predictionusing a plurality of signals obtained immediately before the prediction,FIG. 15B is a graph showing the case of performing the shake predictionusing the adaptive filter shown in FIG. 13, and FIG. 15C is a graphshowing the case of not performing the shake prediction. Note that, inFIGS. 15A to 15C, the horizontal axis indicates the time, and thevertical axis indicates the actual camera shake amount, the detectedcamera shake amount, the image stabilization amount, or the remainingshake amount after stabilization.

Time t1 is the timing of receiving the signal S2 (e.g., the timing ofstarting exposure), and at the time before the time t1, the imagestabilization processing is being performed, i.e., the signal S1 isoutput. At the time after time t2 after the exposure, the imagestabilization processing is not performed, i.e., the signal S0 isoutput. The period from the time t1 to the time t2 in which the signalS2 is output is the time in which the exposure is being performed, andthe shake detection signal based on the image signal output from theimage sensor 6 cannot be obtained. For this reason, in this period, theimage stabilization control based on the prediction value is performed.Furthermore, in FIGS. 15A to 15C, 263, 264, and 265 denote the magnitudeof the remaining shake amount after stabilization in the case of usingthe respective methods.

In FIGS. 15A to 15C, thick solid lines denote the actual camera shakeamount applied to the camera system, and alternate long and short dashlines denote the detected camera shake amount u(m) or the predictionvalue y(m) that is output by the prediction signal generation unit 223when the exposure is being performed. Broken lines denote the imagestabilization amount, and alternate long and two short dashes linesdenote the shake detection signal, which is the remaining shake amountafter stabilization. Note that the camera shake amount u(m) issubstantially equal to the actual camera shake amount while the shake iscorrectly being detected, and indicates, during the exposure, theprediction value y(m) and corresponds to the signal output from theswitch 225 in FIG. 12. As described using FIG. 14, if the signal S1 isoutput, the stabilization amount of the image stabilization lens isobtained based on the camera shake amount u(m), which is obtained byadding the shake detection signal to the amount of change of the angleof view, as mentioned above. The image stabilization amount is aresponse to shake control input regarding which mechanical followingcharacteristics are considered, and corresponds to the imagestabilization amount achieved via the control unit 226 in FIG. 12. Theremaining shake after stabilization is the difference between the actualcamera shake amount and the image stabilization amount. If the remainingshake after stabilization is small during the exposure, the shake can beconsidered to be small.

A case of performing the linear prediction will be described using FIG.15A. If the signal S2 is received at the time t1, a plurality of camerashake amounts u(m) that have been obtained immediately before receivingthe signal S2 are linearly approximated, and the obtained line isextended while keeping the tilt thereof to predict the camera shakeamount during the exposure. In FIG. 15A, the alternate long and dot dashline of the shake control input appears to be linear on the actualcamera shake amount between the time t1 and the time t2 during theexposure. This is the linearly predicted value. Thereafter, upon theexposure being completed, the signal S0 is input at the time t2, and theswitch 227 in FIG. 12 is switched and is grounded to the ground portion228. Thereby, the image stabilization lens is moved to the referenceposition at which the center thereof coincides with the optical axis,and the angle of view of the object that is incident on the image sensor6 changes, but the image that is being shot is not affected.

Here, considering the shake that has occurred during the exposure inFIG. 15A, the amount denoted by 263 in FIG. 15A is the magnitude of theremaining shake amount, and it can be understood that the influence ofthe shake is suppressed compared with the case of not performing shakeprediction, which will be described using FIG. 15C. The shake that canbe approximated by a line as described in FIG. 15A tends to occur when astanding human performs image shooting, for example. When standing, thebody often swings to the left and right very slowly. This frequency(e.g., 0.1 Hz) is lower than the frequency of a camera shake that occursdue to a hand or an arm, and the magnitude of the swing is large. Forthis reason, considering a general exposure period (about 1/60s), theshake can be approximated by a line.

Next, the case of performing prediction using the adaptive filter willbe described with reference to FIG. 15B. If the signal S2 is received atthe time t1, the prediction value y(n+1) of the camera shake amountduring the exposure is generated using the prediction signal generationunit 223. As described using FIG. 13, the recursive calculation isperformed to generate the prediction value during the exposure, and thegenerated prediction value is used in the control. Almost correctprediction is performed with the simple waveform shown in FIG. 15B, andthe shake prediction value roughly coincides with the actual shake inthe period from the time t1 to the time t2 during the exposure as well.Thereafter, upon the exposure being completed, the signal S0 is input atthe time t2, and the switch 227 in FIG. 12 is switched and is groundedto the ground portion 228. Thereby, the image stabilization lens ismoved to the reference position, and the angle of view of the objectthat is incident on the image sensor 6 changes, but the image that isbeing shot is not affected.

Here, considering the shake that has occurred during the exposure inFIG. 15B, the amount denoted by 264 in FIG. 15B is the magnitude of theremaining shake amount, and it can be understood that the shake isbetter suppressed compared with the case of not performing the shakeprediction, which will be described using FIG. 15C. In the case of usingthe adaptive filter, it is conceivable that the prediction signalgeneration unit 223 stores shake spectrum information in the form of thefilter coefficients. For this reason, if appropriate information forupdating the filter coefficients in the prediction signal generationunit 223 can be obtained, the accurate prediction is possible as shownin FIG. 15B. Note that although FIG. 15B shows the case of a simplewaveform, the actual camera shake also need only be considered to besuperposition of a plurality of simple waveforms, and accordingly, thereare many situations where accurate prediction can be performed using theadaptive filter.

Next, a case of not performing the shake prediction during the exposurewill be described with reference to FIG. 15C. If the signal S2 isreceived at the time t1, the image stabilization is stopped. In FIG.15C, the image stabilization amount is zero in the period from the timet1 to the time t2. Of course the remaining shake amount afterstabilization during the exposure is the actual camera shake amountitself. Here, considering the shake that has occurred during theexposure in FIG. 15C, the amount denoted by 265 in FIG. 15C is themagnitude of the remaining shake, and it can be understood that a largeremaining shake is occurring.

As described above, according to the fifth embodiment, a system forperforming prediction processing using the post-stabilization remainingshake signal can be established in the system for performing the imagestabilization using the signal of the image sensor. As a result,improvement of picture quality can be achieved while providing asequence appropriate for image shooting.

Note that although the above example has described the case ofperforming the image stabilization control by driving the imagestabilization lens included in the imaging optical system 3, the imagestabilization control can also be performed by driving the image sensor6.

Sixth Embodiment

Next, a sixth embodiment of the present invention will be described. Thefifth embodiment has described the image stabilization, whereas thesixth embodiment will describe a case of applying the predictionprocessing to focus adjustment. Note that since the basic configurationof the camera system is similar to that described in the fifthembodiment, a description thereof will be omitted. In the fifthembodiment, since the remaining shake after stabilization needs to bedetected from the image signal obtained from the image sensor 6, theshake detection based on the optical flow detection is performed.Meanwhile, in the sixth embodiment, the focus state is detected from theimage signal obtained from the image sensor 6. Note that the focus stateis detected by the image processing unit 7.

FIGS. 16A to 16C are diagrams showing a configuration of the imagesensor 6 according to the sixth embodiment. The image sensor 6 accordingto the sixth embodiment is configured such that a plurality ofphotoelectric conversion units are assigned to one microlens, and suchmicrolenses are arranged in an array and constitute a microlens array120.

FIG. 16A is a schematic view of the microlens array 120 in the imagesensor 6 as viewed from the front and a side. As shown in FIG. 16A, themicrolens array 120 is provided on the image sensor 6 so as to cover theimage sensor 6 as viewed from the front, and the front principal pointof the microlens array 120 is arranged near the image forming plane ofthe imaging optical system 3.

FIG. 16B is a schematic view showing correspondence of 2×2 microlensesin the microlens array 120 as viewed from the front to a plurality ofphotoelectric conversion units. Two photoelectric conversion units 121 aand 121 b are associated with each microlens 120 a that constitutes themicrolens array 120.

FIG. 16C is a diagram showing that the plurality of photoelectricconversion units 121 a and 121 b provided below the microlens 120 a areassociated with different regions of an exit pupil region of the imagingoptical system 3 by the microlens 120 a. FIG. 16C is a cross-sectionalview of the image sensor 6 taken so as to include the optical axis ofthe microlens 120 a, with the longitudinal direction of the sensor beingthe lateral direction of the diagram. Note that, in practice, if thedirection of the photoelectric conversion units 121 a and 121 b shown inthe lower part of FIG. 16C is aligned with the direction of the exitpupil plane, the exit pupil plane exits on a plane that is normal to thepage of FIG. 16C, but the direction of the exit pupil plane is rotatedso as to be shown on the plane of the page of FIG. 16C for the sake ofthe description.

As described in FIG. 16C, the photoelectric conversion units 121 a and121 b are designed so as to be conjugate respectively to specificregions 131 and 132 on the exit pupil plane of the imaging opticalsystem 3 through the microlens 120 a. That is to say, a light beam thathas passed through a partial region 131 of the exit pupil of the imagingoptical system 3 is incident on the photoelectric conversion unit 121 a.A light beam that has passed through a partial region 132 of the exitpupil of the imaging optical system 3 is incident on the photoelectricconversion unit 121 b.

Thus, since the image sensor 6 according to the sixth embodiment canindependently acquire an image signal corresponding to light beams thathave passed through different regions of the exit pupil, a focus statecan be detected based on the principle of so-called phase difference AF.Specifically, image signals obtained from the photoelectric conversionunits 121 a corresponding to the region 131, of the plurality of thephotoelectric conversion units 121 a and 121 b provided below eachmicrolens 120 a, are collected to form one image. Similarly, the imagesignals obtained from the photoelectric conversion units 121 bcorresponding to the region 132 are collected to form another image. Afocus state is detected based on the phase difference by performingknown correlation calculation based on a pair of image signals that arethus obtained.

Then, in a state where the signal S1 is input, in a so-called servo AFmode of continuously performing AF control, a focus state is detected,and the detected focus state is added to the amount of movement of thefocus lens detected by the position sensor 15. Processing can beperformed similarly as with the camera shake amount described in thefifth embodiment, using the value (combined signal) that is thusobtained by addition.

Seventh Embodiment

Next, a seventh embodiment of the present invention will be described.The above fifth embodiment has described the method for performing theprediction after adding the shake detection signal to the amount ofchange of the angle of view that is based on the lens position signalobtained from the position sensor 15, whereas the seventh embodimentwill describe a case of performing control based only on the shakedetection signal.

FIG. 17 is a block diagram showing an exemplary configuration related tothe image stabilization according to the seventh embodiment that isincluded in the camera system control circuit 5. A difference in theconfiguration between FIG. 17 and FIG. 12 lies in that the gainadjustment unit 230 for processing the lens position signal from theposition sensor 15 and the adder 222 are not provided, and that thecontrol of the prediction signal generation unit 223′ using the resetsignal is not performed. With this configuration, not the camera shakeamount but the shake detection signal detected by the shake detectionunit 70 is input to the prediction signal generation unit 223′. Theother elements are similar to those shown in FIG. 12, and accordinglythe same reference numerals will be assigned thereto and a descriptionthereof will be omitted. Note that in the seventh embodiment, not thecamera shake amount but the shake detection signal will be expressed asu(m).

As described in the fifth embodiment with reference to FIG. 14, thecharacteristics of the shake detection signal detected by the shakedetection unit 70 before and after the start of the image stabilizationgreatly change. In the seventh embodiment, the prediction is performedbased on the waveform of the remaining shake amount after stabilizationdenoted by the broken line in FIG. 14. Since the waveform of theremaining shake amount after stabilization can be obtained only byoperating the image stabilization lens, in the seventh embodiment, thefilter coefficients are updated while the signal S1 is input, asdescribed later using FIG. 18.

Furthermore, in the seventh embodiment, lens control is performed basedon the shake detection signal. This is a control method that is alsocalled a so-called zero method or the like, where if the remaining shakeoccurs, the position of the image stabilization lens is adjusted so asto cancel the remaining shake. That is to say, if a shake occurs, thisshake is detected by the shake detection unit 70, and stabilization isimmediately performed (in a period that is roughly equal to a feedbackcontrol period).

Note that the switching of the switches 225 and 227 is performedsimilarly as in the fifth embodiment. Meanwhile, if the signal S0 isinput, the image stabilization has stopped, and the filter coefficientsare not updated in the prediction signal generation unit 223′.Thereafter, if the signal S1 is input, the image stabilization isperformed based on the shake detection signal, and the filtercoefficients are updated in the prediction signal generation unit 223′.

FIG. 18 is a diagram showing a configuration of the prediction signalgeneration unit 223′ according to the seventh embodiment. A differencefrom the prediction signal generation unit 223 according to the fifthembodiment shown in FIG. 13 lies only in a timing signal for connectionto the ground portion 236. In the fifth embodiment, the Reset signal isreceived and the filter coefficients are initialized. On the other hand,in the seventh embodiment, a correct signal is obtained after the signalS1 is input, and therefore, the filter coefficients are set to be in aninitialized state before the signal S1 is input. Upon entering a statewhere the signal S1 is input, the switch 237 operates and is connectedto the output from the adaptive algorithm processing unit 235, and thefilter coefficients 232 are updated. This configuration enables thefilter coefficients 232 to be updated based on the remaining shakeamount after stabilization denoted by the broken line in FIG. 14.Furthermore, if the signal S2 is input, the image stabilization lens iscontrolled based on the prediction value y(n+1) that is computed usingthe filter coefficients 232 and is output from the prediction signalgeneration unit 223′.

FIGS. 19A to 19C are diagrams for illustrating the camera shake amountand the remaining shake amount after stabilization in the case ofperforming the prediction processing and in the case of not performingthe prediction processing. In order to facilitate understanding of thedescription, it is assumed also in FIGS. 19A to 19C that the actualshake applied to the camera system is a simple sine wave. Furthermore,similarly to FIGS. 15A to 15C, it is assumed here that the signal S0,the signal S1, and the signal S2 are switched by an operation made on ashutter button (not shown) when shooting a still image. Specifically,the signal S0 is output in a state where the shutter button is notpressed, the signal S1 is output in a state where the shutter button ishalf-pressed and an instruction to prepare for shooting has been given,and the signal S2 is output in a state where the shutter button isfull-pressed and an instruction to recording an image has been given.

FIG. 19A is a graph showing the case of performing the linear predictionusing a plurality of signals obtained immediately before the prediction,FIG. 19B is a graph showing the case of performing the shake predictionusing the adaptive filter shown in FIG. 18, and FIG. 19C is a graphshowing the case of not performing the shake prediction. Note that, inFIGS. 19A to 19C, the horizontal axis indicates the time, and thevertical axis indicates the actual camera shake amount, the detectedcamera shake amount, the image stabilization amount, or the remainingshake amount after stabilization. Note that FIG. 19C is similar to FIG.15C, and therefore, a description thereof will be omitted here.

Time t11 is the timing of receiving the signal S2 (e.g., the timing ofstarting exposure), and at the time before the time t11, the imagestabilization processing is being performed, i.e., the signal S1 isoutput. At the time after time t12 after the exposure, the imagestabilization processing is not performed, i.e., the signal S0 isoutput. The period from the time t11 to the time t12 in which the signalS2 is output is the time in which the exposure is being performed, andthe shake detection signal based on the image signal output from theimage sensor 6 cannot be obtained. For this reason, in this period, theimage stabilization control based on the prediction value is performed.In FIGS. 19A to 19C, 163, 164, and 165 denote the magnitudes of theremaining shake amount after stabilization in the case of the respectivemethods.

In FIGS. 19A to 19C, thick solid lines denote the actual camera shakeamount applied to the camera system, and alternate long and short dashlines denote the shake detection signal, or the prediction value y(m)that is output by the prediction signal generation unit 223′ during theexposure. Broken lines denote the image stabilization amount, andalternate long and two short dashes lines denote the remaining shakeamount after stabilization. Control is performed using a so-called zeromethod in which appropriate phase compensation and gain compensation areperformed on the shake detection signal to drive the image stabilizationlens. The amount of the stabilization performed by the imagestabilization lens is a response to shake control input regarding whichmechanical following characteristics and the like are considered, andcorresponds to the image stabilization amount achieved via the controlunit 226 in FIG. 17. The shake detection signal, which is the remainingshake after stabilization, indicates the difference between the actualcamera shake amount and the amount of the stabilization performed by theimage stabilization lens. If the remaining shake after stabilizationduring the exposure is small, the actual shake can be considered to besmall.

A case of performing the linear prediction will be described using FIG.19A. If the signal S2 is received at the time t11, a plurality of shakedetection signals u(m) that have been obtained immediately beforereceiving the signal S2 are linearly approximated, and the obtained lineis extended while keeping the tilt thereof to predict the shakedetection signal during the exposure. In FIG. 19A, the alternate longand short dash line indicating the prediction value of the shakedetection signal appears to be linear and slightly deviate from the lineindicating the actual camera shake amount in the period from the timet11 to the time t12 during the exposure. This is the linearly predictedvalue. Considering the shake that has occurred during the exposure inFIG. 19A, the amount denoted by 163 in FIG. 19A is the magnitude of theshake, and it can be understood that the influence of the shake issuppressed compared with the case of not performing the shake predictionshown in FIG. 19C.

Next, the case of performing the prediction using the adaptive filterwill be described with reference to FIG. 19B. If the signal S2 isreceived at the time t11, the prediction value y(n+1) of the shakedetection signal during the exposure is generated using the predictionsignal generation unit 223′. In the example in FIGS. 19A to 19C,similarly as in the case of linear prediction, the alternate long andshort dash line of the shake control input appears to be linear so as tobe slightly deviate from the line of the remaining shake afterstabilization in the period from the time t11 to the time t12 during theexposure. Considering the shake that has occurred during the exposure inFIG. 19B, the amount denoted by 164 in FIG. 19B is the magnitude of theshake, and it can be understood that the influence of the shake issuppressed compared with the case of not performing the shake predictionshown in FIG. 19C.

As described above, according to the seventh embodiment, a system forperforming prediction processing using the post-stabilization remainingshake signal can be established in the system for performing the imagestabilization using the signal of the image sensor. As a result,improvement of picture quality can be achieved while providing asequence appropriate for shooting.

Note that, although the above embodiments have described the cases ofapplying the present invention to the camera system constituted by thecamera body 1′ and the lens unit 2′, it is needless to say that theinvention is also applicable to an image capturing apparatus in which acamera body and a lens unit are integrally configured.

Other Embodiments

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiments and/or that includes one or morecircuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiments, and by a method performed by the computer of the system orapparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiments and/or controlling theone or more circuits to perform the functions of one or more of theabove-described embodiments. The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application Nos.2015-116022 and 2015-116024, both filed on Jun. 8, 2015 which are herebyincorporated by reference herein in their entirety.

What is claimed is:
 1. An image processing apparatus comprising: aprocessor and a memory storing therein program instructions which, whenexecuted, cause the processor to operate as following units: a detectionunit that detects moving information using a plurality of images outputfrom an image sensor; a prediction unit that predicts moving informationcorresponding to a first image that is captured after a first imagegroup based on moving information detected using a plurality of imagesincluded in the first image group; and a calculation unit thatcalculates a coefficient to be used in prediction processing by theprediction unit, wherein the calculation unit calculates a coefficientto be used in the prediction processing for predicting movinginformation corresponding to a second image captured after the firstimage using the moving information corresponding to the first image thatis predetected by the prediction unit and moving informationcorresponding to the first image detected by the detection unit.
 2. Theimage processing apparatus according to claim 1, wherein the predictionunit predicts the moving information corresponding to the second imageusing moving information detected using a plurality of images includedin a second image group that includes the first image and thecoefficient for the second image calculated by the calculation unit. 3.The image processing apparatus according to claim 1, wherein thedetection unit detects moving information of a main object.
 4. The imageprocessing apparatus according to claim 1, wherein the prediction unitperform the prediction processing during live view display or movingimage shooting.
 5. An image capturing apparatus comprising: a processorand a memory storing therein program instructions which, when executed,cause the processor to operate as following units: a detection unit thatdetects moving information using a plurality of images output from animage sensor; a prediction unit that predicts moving informationcorresponding to a first image that is captured after a first imagegroup based on moving information detected using a plurality of imagesincluded in the first image group; a calculation unit that calculates acoefficient to be used in prediction processing by the prediction unit;and a driving unit that drives a correction lens or the image sensorbased on the moving information predicted by the prediction unit,wherein the calculation unit calculates a coefficient to be used in theprediction processing for predicting moving information corresponding toa second image captured after the first image using the movinginformation corresponding to the first image that is predetected by theprediction unit and moving information corresponding to the first imagedetected by the detection unit.
 6. An image processing methodcomprising: detecting moving information using a plurality of imagesincluded in a first image group output from an image sensor; calculatinga coefficient to be used in prediction processing; performing theprediction processing for predicting moving information corresponding toa first image that is captured after the first image group based on thedetected moving information; detecting moving information correspondingto the first image; and calculating a coefficient to be used inprediction processing for predicting moving information corresponding toa second image captured after the first image using the predicted movinginformation corresponding to the first image and the detected movinginformation corresponding to the first image.